transformers

The Transformers repository is a natural language processing (NLP) library that provides pre-trained models and utilities for tasks such as text classification, question answering, language generation, and computer vision. Engineers can use it to solve real-world problems in natural language understanding and generation, as well as multimodal tasks involving text, images, and audio.

The most important parts of the repository are:

1. Model Implementations (…/models): This directory contains implementations of various pre-trained models like BERT, GPT, T5, CLIP, and models such as Gemma2, LLaVa-NeXT-Video, and Qwen2VL. These models can be fine-tuned for specific NLP and computer vision tasks.

2. Pipeline Functionality (…/pipelines): The pipeline module provides a user-friendly interface for using pre-trained models. It encapsulates task-specific logic, including input preprocessing, model inference, and output postprocessing. This allows users to easily integrate Transformers capabilities into their applications. For more details, see the Pipelines section.

3. Utility Functions (…/utils): This directory contains utility functions and classes used throughout the library, covering functionality such as compatibility management, file handling, and tensor manipulation. These utilities are essential for the library's operation and extensibility.

The Transformers library is built on top of machine learning frameworks like PyTorch, TensorFlow, and JAX. It leverages transformer-based models, which have become standard for many NLP and computer vision tasks. The library provides a consistent interface for working with these models, allowing users to fine-tune and evaluate them on custom datasets.

Key algorithms and technologies:

• Attention mechanisms: The library implements various attention mechanisms, including scaled dot-product attention and more efficient variants like Flash Attention 2.
• Tokenization: Different tokenization strategies are employed for processing text inputs, including subword tokenization methods.
• Model architectures: The library supports encoder-only, decoder-only, and encoder-decoder architectures, as well as vision-language models.

Key design choices:

• Modular architecture: The library is organized into separate modules for models, pipelines, and utilities, enhancing maintainability and extensibility.
• Framework agnostic: Support for multiple machine learning frameworks allows users to choose their preferred backend.
• Test suite: The repository includes unit tests and integration tests for model implementations. For more information, see the Testing section.
• Documentation: Detailed documentation is provided in the …/source directory, covering configuration, usage, training, and testing.

Model Implementations

References: src/transformers/models, src/transformers/quantizers, docs/source/en/model_doc/mllama.md, docs/source/en/model_doc/omdet-turbo.md, docs/source/en/quantization/compressed_tensors.md, src/transformers/models/idefics3, src/transformers/models/mllama, src/transformers/models/omdet_turbo

The …/models directory contains the implementations of various pre-trained models within the Transformers library. Each subdirectory in this directory corresponds to a specific model and includes the necessary components for working with that model, such as configuration, tokenization, modeling, and utility scripts.

The key functionality provided in this directory includes:

• Configuration Classes: The directory contains configuration classes for various models, such as AlbertConfig, BartConfig, BertConfig, RobertaConfig, and PixtralVisionConfig. These classes are responsible for storing the configuration parameters of the corresponding models.

• Tokenization: The directory includes tokenizer implementations for models like AlbertTokenizer, BartTokenizer, BertTokenizer, RobertaTokenizer, and PixtralProcessor.

• Model Implementations: The directory provides the core PyTorch, TensorFlow, and Flax implementations of the various models, such as AlbertModel, BartModel, BertModel, RobertaModel, and PixtralVisionModel. These model classes encapsulate the model architecture and provide methods for forward passes and feature extraction.

• Utility Scripts: The directory also includes utility scripts for converting pre-trained checkpoints from other libraries (e.g., TensorFlow) to a format compatible with the Transformers library. For example, the convert_albert_original_tf_checkpoint_to_pytorch.py script can be used to convert a TensorFlow checkpoint of the ALBERT model to a PyTorch model.

Some key design choices and implementation details in this directory include:

• The use of configuration classes to provide a flexible and extensible way to customize the various model parameters, such as the number of layers, attention heads, and hidden sizes.
• The separation of concerns between the tokenization, configuration, and model implementation, which promotes modularity and reusability.
• The inclusion of utility scripts for converting pre-trained checkpoints from other libraries, which simplifies the process of using pre-trained models in the Transformers library.

The directory also includes implementations for multimodal models that combine text and image processing:

• LlavaForConditionalGeneration: This model in …/modeling_llava.py handles the integration of image features with text embeddings. It includes methods for merging input IDs with image features and preparing inputs for generation tasks.

• LlavaNextForConditionalGeneration: Found in …/modeling_llava_next.py, this model supports handling images of any resolution. It includes functions for calculating image patch grids, unpadding images, and merging image features with text inputs.

• LlavaNextVideoForConditionalGeneration: Located in …/modeling_llava_next_video.py, this model extends the LLAVA-NeXT architecture to handle video inputs. It includes methods for pooling image features and integrating them with the language model.

• VideoLlavaForConditionalGeneration: This model in …/modeling_video_llava.py is designed to handle both image and video inputs in conjunction with text. It includes methods for processing inputs that may contain more than the maximum allowed number of image or video tokens.

• VipLlavaForConditionalGeneration: Found in …/modeling_vipllava.py, this model focuses on integrating visual information with text inputs. It includes methods for merging image features with text embeddings and handling cases where the number of image tokens exceeds the expected length.

• PixtralVisionModel: This model in …/modeling_pixtral.py is designed to process image inputs. It includes components such as PixtralRotaryEmbedding, PixtralAttention, and PixtralTransformer to handle image patches and apply transformations.

These multimodal models demonstrate the library's capability to process and generate content based on both textual and visual inputs.

For more information on the specific model implementations, please refer to the following sections:

ALBERT BART

Model Architectures and Implementations

References: src/transformers/models/align, src/transformers/models/altclip, src/transformers/models/auto, src/transformers/models/blip, src/transformers/models/blip_2, src/transformers/models/bridgetower, src/transformers/models/chameleon, src/transformers/models/chinese_clip, src/transformers/models/clip, src/transformers/models/clipseg, src/transformers/models/cohere, src/transformers/models/convbert, src/transformers/models/convnextv2, src/transformers/models/deberta, src/transformers/models/distilbert, src/transformers/models/electra, src/transformers/models/funnel, src/transformers/models/fuyu, src/transformers/models/gemma2, src/transformers/models/git, src/transformers/models/idefics2, src/transformers/models/imagegpt, src/transformers/models/instructblip, src/transformers/models/instructblipvideo

Transformers models for object detection and segmentation, such as ConditionalDetrModel and DeformableDetrModel, leverage a convolutional backbone combined with a transformer encoder-decoder architecture. The ConditionalDetrModel introduces ConditionalDetrFrozenBatchNorm2d to replace standard batch normalization layers, aiming to stabilize training by fixing batch statistics and affine parameters. The DeformableDetrModel features a DeformableDetrMultiscaleDeformableAttention mechanism, allowing the model to focus on a small set of key sampling points around a reference, enhancing its ability to handle objects of various sizes.

• ConditionalDetrModel and DeformableDetrModel both utilize a convolutional backbone to extract multi-scale feature maps, which are then processed by the transformer encoder-decoder to generate object detection predictions.
• Position embeddings play a crucial role in these models, with ConditionalDetrSinePositionEmbedding and DeformableDetrLearnedPositionEmbedding providing spatial context to the transformer layers.
• The GroundingDinoModel integrates vision and text features using a bi-directional multi-head attention mechanism, GroundingDinoBiMultiHeadAttention, to perform object detection and grounding tasks.
• The RTDetrModel focuses on real-time performance by combining a feature pyramid network with a transformer-based encoder, RTDetrHybridEncoder, and a transformer-based decoder, RTDetrDecoder.

For speech recognition tasks, the SEWDModel and its variants like SEWDForCTC and SEWDForSequenceClassification provide a framework for processing raw audio waveforms and performing tasks like keyword spotting and speech-to-text conversion.

• SEWDModel processes audio input through a feature encoder and transformer encoder, outputting hidden states for downstream tasks.
• SEWDForCTC adds a CTC head to the base model for sequence-to-sequence learning, while SEWDForSequenceClassification includes a sequence classification head for tasks like keyword spotting.

In the realm of table-based models, TapasModel and its derivatives such as TapasForQuestionAnswering and TapasForSequenceClassification are designed to understand the structure and content of tabular data.

• TapasModel incorporates token type embeddings to capture the tabular structure, extending BERT's capabilities to handle tables.
• TapasForQuestionAnswering specializes in selecting relevant table cells and performing optional aggregation for answering questions posed in natural language.

The …/modeling_idefics2.py file introduces the Idefics2 model, which includes several new classes and functionalities:

• Idefics2BaseModelOutputWithPast and Idefics2CausalLMOutputWithPast are dataclasses representing the output of the Idefics2 model.
• Idefics2VisionEmbeddings enables images of variable resolution to be processed by the model.
• Attention mechanisms such as Idefics2VisionAttention and Idefics2VisionFlashAttention2 are implemented for image feature extraction.
• MLP layers are handled by Idefics2VisionMLP and Idefics2MLP.
• Idefics2MultiheadAttentionPoolingHead is used for multihead attention pooling.
• The encoder and transformer architecture of Idefics2 is constructed using Idefics2EncoderLayer, Idefics2Encoder, and Idefics2VisionTransformer.
• repeat_kv function is utilized to repeat key and value tensors.
• Idefics2RMSNorm is used for layer normalization within the model.
• Perceiver attention mechanisms are implemented through Idefics2PerceiverAttention and Idefics2PerceiverFlashAttention2.
• Idefics2PerceiverLayer and Idefics2PerceiverResampler are used for the perceiver layer and resampler, respectively.
• Idefics2Connector connects the vision and text components of the Idefics2 model.

For more details on the pipeline functionality and utility functions used across these models, refer to the sections Pipelines and Utilities.

Attention Mechanisms and Caching

References: src/transformers/models

The Transformers library implements various attention mechanisms to optimize performance across different models:

• Flash Attention: A memory-efficient attention algorithm that reduces memory usage and improves speed. Implemented in classes like BartFlashAttention2 and BartSdpaAttention for the BART model, BertSdpaSelfAttention for BERT, and GPTNeoXFlashAttention2 for GPT-NeoX, among others.

• Scaled Dot-Product Attention (SDPA): An efficient attention mechanism used in models like BERT and GPT-2. The BertSdpaSelfAttention class provides a specialized implementation for BERT, while GPTNeoXSdpaAttention is used in GPT-NeoX.

• Neighborhood Attention: Used in the DiNAT model, this mechanism computes attention weights based on spatial relationships between input patches. The NeighborhoodAttention class implements this functionality.

• Multi-scale Deformable Attention: Utilized in the Mask2Former model's pixel decoder, implemented in the multi_scale_deformable_attention() function.

Caching strategies are employed to improve inference performance:

• Key-Value Caching: Models like GPT-2 use caching to store previously computed key and value tensors, reducing redundant computations during autoregressive generation.

• Static KV Cache: An optimization technique mentioned in the LLM optimization documentation, which pre-allocates memory for key and value tensors to avoid repeated memory allocations.

These attention mechanisms and caching strategies are crucial for improving the efficiency and performance of transformer-based models, especially for tasks involving long sequences or real-time generation.

Tokenization and Vocabulary Management

References: src/transformers/models/bloom

The BloomTokenizerFast class, located in …/tokenization_bloom_fast.py, is integral to the BLOOM model's ability to process text. It manages the complexities of tokenization, ensuring that text inputs are correctly transformed into a format suitable for the model. One of its key features is the handling of prefix spaces, which is crucial for models trained on data that starts with a space. This tokenizer also provides methods like _batch_encode_plus() and _encode_plus() that are tailored to handle pre-tokenized inputs efficiently.

Tokenization in BLOOM involves managing merges and vocabularies, which are essential for understanding and generating human-like text. The tokenizer's save_vocabulary() method allows for the persistence of the tokenizer's vocabulary, facilitating the reuse and sharing of tokenization schemes.

The BloomConfig class in …/configuration_bloom.py supports the tokenizer by defining model-specific parameters, such as the vocabulary size. This configuration is critical when initializing the tokenizer to ensure consistency with the model's expected input structure.

For users interested in exporting the BLOOM model to ONNX format, the BloomOnnxConfig class provides necessary configurations and methods to generate dummy inputs, which are essential for the ONNX export process.

In the context of the BLOOM model's implementation in Flax, as seen in …/modeling_flax_bloom.py, the tokenizer's role is equally important. It ensures that the text inputs are compatible with the model's architecture, which includes attention mechanisms and MLP components that process the tokenized input.

The tokenizer's implementation is designed to be fast and efficient, leveraging the capabilities of the tokenizers library. This design choice underscores the importance of performance and scalability in NLP tasks, where processing large volumes of text quickly is often required.

For further details on the BLOOM model's architecture and functionalities, refer to the sections Model Implementations and Quantization with Compressed Tensors.

Model Configuration and Processing

References: src/transformers/models/auto, src/transformers/models/align, src/transformers/models/chameleon, src/transformers/models/fuyu, src/transformers/models/instructblip, src/transformers/models/instructblipvideo

Managing configurations and processing for model inputs and outputs is facilitated by a variety of classes designed to handle the complexities of multimodal models, which often require the integration of both image and text data. The AutoConfig class, located in …/configuration_auto.py, serves as a dynamic entry point for instantiating configuration objects specific to a given model. It leverages mappings like CONFIG_MAPPING_NAMES and MODEL_NAMES_MAPPING to associate model identifiers with their respective configuration classes.

For image processing, the AutoImageProcessor class in …/image_processing_auto.py automatically selects and instantiates the appropriate image processor based on the model type. It uses the IMAGE_PROCESSOR_MAPPING_NAMES dictionary to map model types to image processor classes, ensuring that images are preprocessed correctly for the model in question.

Text processing is similarly streamlined through the AutoTokenizer class found in …/tokenization_auto.py. This class determines the correct tokenizer to use based on the model's configuration or type, facilitating the handling of text inputs for various pre-trained models.

The AutoProcessor class, detailed in …/processing_auto.py, provides a unified interface for processing both text and image inputs. It intelligently combines the functionalities of tokenizers, image processors, or feature extractors, depending on the model's requirements, to prepare data for training or inference.

For models that require handling of both text and image data, such as those in the …/align directory, specialized processing classes like AlignProcessor are used. This processor wraps both EfficientNetImageProcessor and BertTokenizer to handle mixed inputs, replacing image tokens with sequences representing the image.

In the case of the Chameleon model, located in …/chameleon, the ChameleonProcessor class processes mixed text and image inputs, utilizing the ChameleonImageProcessor and LlamaTokenizerFast to provide a seamless preprocessing experience for this image-to-text generation model.

These configuration and processing classes are critical for setting up models to handle the diverse data types encountered in multimodal tasks. By abstracting the preprocessing steps and providing a consistent interface, they significantly simplify the workflow for users working with the Transformers library.

Model Renaming and Refactoring

References: src/transformers/models/pixtral/__init__.py, src/transformers/models/pixtral/configuration_pixtral.py, src/transformers/models/pixtral/modeling_pixtral.py

In the context of model development within the Hugging Face Transformers library, the Pixtral model has undergone a renaming and refactoring process to align its nomenclature with its enhanced capabilities and to streamline its use for inference and fine-tuning. The Pixtral model, encapsulated within the …/pixtral directory, has been structured to facilitate ease of use and integration into various workflows.

• The PixtralVisionConfig class provides a comprehensive set of parameters to configure the Pixtral vision encoder model, including attributes like hidden_size, num_hidden_layers, and num_attention_heads. This allows users to tailor the model's architecture to their specific needs for tasks involving vision processing.
• The PixtralVisionModel serves as the primary interface for the Pixtral vision encoder, incorporating a Transformer-based architecture that leverages custom layers and attention mechanisms. The model is designed to be flexible, supporting various vision-related tasks.
• The PixtralProcessor and PixtralImageProcessor classes offer a unified approach to preprocessing, ensuring that both image and text data are appropriately handled before being fed into the model. This simplifies the process of preparing data for the Pixtral model, whether for fine-tuning or inference.
• The refactoring efforts have also included the implementation of specialized components such as PixtralRotaryEmbedding and PixtralAttention, which are integral to the model's ability to understand and process visual information with positional context.

The renaming and refactoring of the Pixtral model reflect a strategic move to optimize the model's performance and usability. By focusing on these aspects, the Transformers library continues to provide robust and efficient solutions for a wide range of machine learning tasks involving vision and language.

Attention Implementation Updates

References: src/transformers/models/m2m_100/modeling_m2m_100.py

The …/modeling_m2m_100.py introduces new attention classes and updates to enhance the M2M-100 model's capabilities. The file includes:

• M2M100Attention class, which is the foundational attention mechanism for the model.
• Variants like M2M100FlashAttention2 and M2M100SdpaAttention that offer alternative attention mechanisms, providing flexibility in handling different input formats and computational efficiency.
• The M2M100Encoder and M2M100Decoder classes utilize these attention mechanisms to process multilingual inputs effectively.

The attention mechanism updates are designed to support a variety of input formats, ensuring that the model can handle the complexities of multilingual machine translation. Additionally, the implementation provides warnings for unsupported features, guiding users when configuring the model for specific tasks.

The M2M100ForConditionalGeneration class extends the model's functionality, enabling language modeling tasks such as summarization. This is achieved by integrating the updated attention mechanisms into the model's architecture, allowing it to generate text conditioned on the input.

Utility functions like shift_tokens_right() and create_position_ids_from_input_ids() support the attention mechanism's functionality by preparing inputs for the model's encoder and decoder. The M2M100ScaledWordEmbedding is another utility that scales word embeddings, which is crucial for the model's performance.

For more details on the model's architecture and implementations for specific tasks, refer to the sections on Model Architectures and Implementations and Text Generation Enhancements.

Pipelines

References: src/transformers/pipelines

The Transformers library provides a set of pipelines for natural language processing (NLP) and computer vision (CV) tasks, allowing users to leverage pre-trained models for their applications. These pipelines abstract the details of model loading, preprocessing, and postprocessing, providing an interface for performing tasks such as text classification, question answering, image classification, and more.

The core functionality of the pipeline system is defined in the …/base.py file, which includes the Pipeline and ChunkPipeline classes. The Pipeline class is the base class for all pipeline implementations, defining the workflow of a pipeline, including the preprocess(), _forward(), and postprocess() methods. The ChunkPipeline class is a subclass of Pipeline that processes inputs in smaller chunks.

The Pipeline class has several key methods:

• build_pipeline_init_args includes documentation for the binary_output parameter.
• save_pretrained handles custom implementations registered in the pipeline, adding their information to the model's configuration before saving.
• ensure_tensor_on_device supports ModelOutput and UserDict objects, in addition to dictionaries and lists.
• check_model_type handles the case where supported_models is a dictionary with tuples of model classes.

The various NLP and CV pipelines are implemented in separate files, each focusing on a specific task or set of related tasks. These pipelines include:

• Text-Based Pipelines
• Image-Based Pipelines
• Audio Pipelines
• Multimodal Pipelines
• Table-Based Pipelines
• Document-Based Pipelines
• Zero-Shot Pipelines
• Feature Extraction Pipelines
• Video Pipelines

Each pipeline class encapsulates task-specific logic, including preprocessing the input, passing it through the model, and postprocessing the output. The pipelines also handle various input formats and parameter validation.

In …/image_classification.py, the torch module is imported at the beginning of the file. The postprocess method ensures that the output tensor's data type is compatible for further processing by converting torch.bfloat16 or torch.float16 tensors to torch.float32 before converting to a NumPy array.

In …/text_generation.py, the _sanitize_parameters method assigns the add_special_tokens parameter to both preprocess_params and a separate variable, and assigns the padding parameter from generate_kwargs. The __call__ method supports string inputs and lists of dictionaries for text_inputs. The preprocess method includes handling of handle_long_generation with the "hole" strategy. The _forward method adjusts max_length and min_length when a prefix is provided. The TextGenerationPipeline class includes a handle_long_generation parameter, which can be set to "hole" to handle long generation by truncating the left side of the input and leaving a gap for the generation to happen. The postprocess method handles the post-processing of the generated sequence, including the handling of the ReturnType enum values.

Pipeline Base Classes

References: src/transformers/pipelines/base.py

The core functionality of the Transformers pipeline system is defined in the file …/base.py. This file includes several key components that form the foundation of all pipeline implementations in the Transformers library.

The primary component is the Pipeline class, which serves as the base class for all pipeline implementations. This class defines the core workflow of a pipeline, including the preprocess(), _forward(), and postprocess() methods. The __call__() method is the main entry point for using a pipeline, handling the overall execution of the pipeline's functionality.

The Pipeline class also includes methods for saving and loading the pipeline. The save_pretrained method handles saving the pipeline configuration, including any custom implementation information. Device placement and tensor management utilities are provided, such as the ensure_tensor_on_device method, which handles ModelOutput and UserDict objects in addition to regular dictionaries and lists.

The ChunkPipeline class, a subclass of Pipeline, is designed to handle inputs that need to be processed in smaller chunks, overriding the run_single() and get_iterator() methods to manage the chunking and recombination of the input and output data.

The file also includes the PipelineDataFormat classes, which support reading and writing data in different formats (CSV, JSON, and piped input/output). These classes handle the loading and saving of data, as well as mapping between dataset columns and pipeline arguments.

The PipelineRegistry class manages the available pipeline tasks and their associated models. It provides methods for registering new pipeline tasks, checking the validity of a task, and retrieving the supported tasks. The registry also handles task aliases and the mapping of translation tasks to the general "translation" task.

Utility functions such as infer_framework_load_model(), infer_framework_from_model(), get_framework(), and get_default_model_and_revision() are included for selecting the appropriate framework (PyTorch or TensorFlow) and model loading. The pad_collate_fn() function is used for padding input tensors for batched processing.

The Pipeline class handles cases where the model is loaded with accelerate, ignoring the device argument and using the device(s) specified by accelerate. The check_model_type method accommodates when the supported_models argument is a dictionary with tuples of model classes. Additionally, the build_pipeline_init_args function includes documentation for the binary_output parameter.

Text-Based Pipelines

References: src/transformers/pipelines/text_classification.py, src/transformers/pipelines/question_answering.py, src/transformers/pipelines/text_generation.py, src/transformers/pipelines/fill_mask.py, src/transformers/pipelines/token_classification.py, src/transformers/pipelines/zero_shot_classification.py, src/transformers/pipelines/text2text_generation.py

The Transformers library provides a set of text-based pipelines that enable users to perform various natural language processing tasks, such as text classification, question answering, and text generation.

The TextClassificationPipeline class is responsible for text classification tasks, allowing users to classify input text as either positive or negative sentiment, or into multiple classes. The pipeline handles preprocessing the input text using the tokenizer, passing the preprocessed input through the model, and postprocessing the model outputs to generate the final classification results.

The QuestionAnsweringPipeline class is used for question-answering tasks. It normalizes the input question and context, preprocesses the data, passes it through the model, and postprocesses the model's output to generate the final answer. The pipeline includes functionality for identifying the most likely answer span within the context, taking into account various constraints such as the maximum answer length and the presence of undesired tokens.

The TextGenerationPipeline class is responsible for generating text using causal language models. It supports various input formats, including strings and lists of dictionaries for chat-based interactions. The pipeline manages the text generation process with features such as:

• Sanitizing parameters in _sanitize_parameters, where add_special_tokens is assigned to both preprocess_params and a separate variable, and padding is directly assigned from generate_kwargs.
• Handling text_inputs in the __call__ method to support different input types.
• Implementing a "hole" strategy in the preprocess method for managing inputs that exceed the model's maximum length.
• Adjusting max_length and min_length in the _forward method based on the prefix_length parameter when a prefix is provided.

The FillMaskPipeline class implements a masked language modeling prediction pipeline, which can be used to predict the missing token in a given text with a masked token. The pipeline supports both PyTorch and TensorFlow-based models that have been trained with a masked language modeling objective, such as BERT.

The TokenClassificationPipeline class is used for named entity recognition (NER) tasks. It supports various aggregation strategies to handle subwords and group related tokens into entities. The pipeline also provides functionality to handle batched inputs, offset mapping, and ignoring specific labels.

The ZeroShotClassificationPipeline class uses a pre-trained NLI model to perform zero-shot classification, where the input sequence and candidate labels are converted into sequence-label pairs and passed through the model.

The Text2TextGenerationPipeline, SummarizationPipeline, and TranslationPipeline classes are used for text-to-text generation tasks, such as summarization and translation, using pre-trained sequence-to-sequence models.

Image-Based Pipelines

References: src/transformers/pipelines/image_classification.py, src/transformers/pipelines/object_detection.py, src/transformers/pipelines/image_segmentation.py, src/transformers/pipelines/image_to_text.py, src/transformers/pipelines/image_to_image.py, src/transformers/pipelines/mask_generation.py, src/transformers/pipelines/depth_estimation.py, src/transformers/pipelines/image_feature_extraction.py

The ImageClassificationPipeline class in …/image_classification.py is responsible for performing image classification tasks using pre-trained models. The key functionality of this class includes:

• Preprocessing: The preprocess() method loads the input image and prepares it for the model using the load_image() function and the image_processor attribute.
• Model Inference: The _forward() method passes the preprocessed input to the model and returns the model outputs.
• Postprocessing: The postprocess() method converts the model outputs into the final classification results. If the output tensor is of type torch.bfloat16 or torch.float16, it is converted to torch.float32 before being converted to a NumPy array. The method then applies the sigmoid() function if the model's problem_type is "single_label_classification" or the number of labels is 1. If the problem_type is "multi_label_classification" or the number of labels is greater than 1, the softmax() function is applied.

The ObjectDetectionPipeline class in …/object_detection.py is responsible for performing object detection tasks using various object detection models. The key functionality includes:

• Preprocessing: The preprocess() method loads the input image(s) and prepares them for the model using the load_image() function and the image_processor.
• Model Inference: The _forward() method passes the preprocessed inputs to the model and returns the model outputs.
• Postprocessing: The postprocess() method converts the model outputs into a list of dictionaries, where each dictionary represents a detected object and contains information about its label, score, and bounding box.

The ImageSegmentationPipeline class in …/image_segmentation.py is responsible for performing image segmentation tasks using various models. The key functionality includes:

• Preprocessing: The preprocess() method loads the input image(s) and prepares them for the model.
• Model Inference: The _forward() method passes the preprocessed inputs to the model and obtains the model outputs.
• Postprocessing: The postprocess() method converts the model outputs into the final segmentation results.

Audio Pipelines

References: src/transformers/pipelines/audio_classification.py, src/transformers/pipelines/automatic_speech_recognition.py, src/transformers/pipelines/zero_shot_audio_classification.py

The Transformers library provides two main pipelines for audio-related tasks: AudioClassificationPipeline and AutomaticSpeechRecognitionPipeline.

The AudioClassificationPipeline is responsible for classifying audio inputs using pre-trained models. It supports various input formats, including raw audio data, audio files, and URLs. The pipeline uses the ffmpeg_read() function to read audio files and convert them to a NumPy array. The AudioClassificationPipeline class handles the preprocessing, model inference, and postprocessing steps. Key features include:

• The preprocess() method handles the different input formats and resamples the audio data to the required sampling rate.
• The _forward() method passes the preprocessed input to the underlying model.
• The postprocess() method takes the model outputs and returns a list of dictionaries with the predicted labels and their corresponding scores.

The AutomaticSpeechRecognitionPipeline is designed to perform automatic speech recognition (ASR) on audio inputs, using pre-trained models and feature extractors. The pipeline supports different types of ASR models, including CTC-based models, sequence-to-sequence models, and models with language model decoding. Key features include:

• The preprocess() method is responsible for preprocessing the input audio, including chunking the audio if necessary, and feeding it to the feature extractor.
• The _forward() method performs the forward pass through the model, handling the differences between CTC-based and sequence-to-sequence models.
• The postprocess() method processes the model outputs, handling the differences between the various model types and returning the final transcription, along with optional timestamps and other metadata.

The ZeroShotAudioClassificationPipeline is a pipeline for zero-shot audio classification, which allows classifying audio inputs without any prior training on the specific task. The key features of this pipeline include:

• The preprocess() method handles the input audio and converts it to a format that can be processed by the model.
• The _forward() method passes the preprocessed inputs through the model and returns the model outputs, including the candidate_labels and the logits.
• The postprocess() method takes the model outputs and converts them into a list of dictionaries, where each dictionary contains the predicted label and its corresponding score.

Multimodal Pipelines

References: src/transformers/pipelines/visual_question_answering.py

The VisualQuestionAnsweringPipeline class, defined in …/visual_question_answering.py, is a pipeline for performing visual question answering tasks. This pipeline combines text and visual information to answer questions about an input image.

The key functionality of this pipeline includes:

• Loading and preprocessing the input image using the load_image() function from …/image_utils.py.
• Passing the image and question input to a pre-trained model for visual question answering, using the MODEL_FOR_VISUAL_QUESTION_ANSWERING_MAPPING_NAMES mapping from …/modeling_auto.py.
• Generating the answer to the question based on the model's output.

The pipeline utilizes several utility functions and classes from other modules, including:

• add_end_docstrings(), is_torch_available(), is_vision_available(), and logging.get_logger() from a utility module.
• KeyDataset from …/pt_utils.py.

The VisualQuestionAnsweringPipeline class provides a user-friendly interface for applying pre-trained visual question answering models to new inputs, abstracting away the details of model loading, preprocessing, and inference.

Document-Based Pipelines

References: src/transformers/pipelines/document_question_answering.py

The DocumentQuestionAnsweringPipeline class in …/document_question_answering.py is responsible for the entire document question answering process, from preprocessing the input to postprocessing the model output.

The key functionality of this pipeline includes:

• Handling different model types (LayoutLM, LayoutLMv2andv3, and VisionEncoderDecoder) and setting the model_type attribute accordingly in the __init__() method.
• Sanitizing the various parameters that can be passed to the pipeline, such as padding, doc_stride, max_question_len, lang, tesseract_config, max_answer_len, max_seq_len, top_k, and handle_impossible_answer, in the _sanitize_parameters() method.
• Preprocessing the input data, which involves loading the image, extracting image features, and optionally using apply_tesseract() to extract words and bounding boxes from the image, in the preprocess() method.
• Passing the preprocessed input through the model using either the model.generate() method (for VisionEncoderDecoder models) or the model() method (for other model types), in the _forward() method.
• Postprocessing the model output to produce the final answer(s), using either postprocess_encoder_decoder_single() or postprocess_extractive_qa() depending on the model type, in the postprocess() method.

The DocumentQuestionAnsweringPipeline provides a user-friendly interface for performing document-based question answering tasks using pre-trained Transformer-based models. It handles the complexities of preprocessing the input, passing it through the model, and postprocessing the output to produce the final answer(s).

Zero-Shot Pipelines

References: src/transformers/pipelines/zero_shot_classification.py, src/transformers/pipelines/zero_shot_image_classification.py, src/transformers/pipelines/zero_shot_object_detection.py

The ZeroShotClassificationPipeline in the Transformers library provides a way to perform text classification tasks without any prior training on the specific task. This pipeline uses a pre-trained Natural Language Inference (NLI) model to determine the relationship between an input sequence and a set of candidate labels, allowing for zero-shot classification.

The key components of the ZeroShotClassificationPipeline implementation are:

• ZeroShotClassificationArgumentHandler: This class is responsible for parsing the input arguments, including the input sequence and the candidate labels, and converting them into the sequence-label pairs required by the NLI model.

  • _parse_labels(): This method processes the input candidate labels and ensures they are in the correct format.
  • __call__(): This method is the main entry point for the argument handler, which takes the input sequence and candidate labels and returns the sequence-label pairs.

• ZeroShotClassificationPipeline:

  • _parse_and_tokenize(): This method tokenizes the input sequence-label pairs and ensures they are in the correct format for the model.
  • _forward(): This method passes the tokenized inputs through the NLI model and returns the model outputs.
  • postprocess(): This method takes the model outputs and computes the final classification results, including the predicted labels and their corresponding scores.

The ZeroShotImageClassificationPipeline in the Transformers library provides a similar zero-shot classification functionality, but for image classification tasks. This pipeline uses the CLIPModel to perform the zero-shot classification, leveraging the model's ability to jointly represent visual and textual information.

The key aspects of the ZeroShotImageClassificationPipeline implementation are:

• The preprocess() method, which loads the input image and creates the necessary input tensors for the model, including the image tensor and the text input tensors.
• The _forward() method, which passes the preprocessed inputs to the CLIPModel and obtains the model outputs.
• The postprocess() method, which converts the model outputs into a format that can be easily consumed by the user, including the predicted labels and their scores.

The ZeroShotObjectDetectionPipeline in the Transformers library provides a pipeline for performing zero-shot object detection using the OwlViTForObjectDetection model. This pipeline allows users to detect objects in an image by providing a set of candidate labels, and it returns the detected objects along with their bounding boxes and confidence scores.

The key components of the ZeroShotObjectDetectionPipeline implementation are:

• The preprocess() method, which loads the input image and tokenizes the candidate labels.
• The _forward() method, which passes the preprocessed inputs to the OwlViTForObjectDetection model and returns the model outputs.
• The postprocess() method, which filters the detected objects based on the provided threshold and top-k parameters, and formats the results as a list of dictionaries containing the label, score, and bounding box for each detected object.

Feature Extraction Pipelines

References: src/transformers/pipelines/feature_extraction.py

The FeatureExtractionPipeline class, defined in …/feature_extraction.py, provides a user-friendly interface for extracting features from transformer models. This pipeline abstracts away the details of tokenization, model forwarding, and postprocessing, allowing users to easily extract features from text or images.

The key functionality of the FeatureExtractionPipeline includes:

• _sanitize_parameters(): This method handles the various parameters that can be passed to the pipeline, such as truncation and tokenize_kwargs.
• preprocess(): This method preprocesses the input data by tokenizing it using the self.tokenizer.
• _forward(): This method passes the preprocessed input through the model and returns the model outputs.
• postprocess(): This method postprocesses the model outputs, extracting the first available tensor and converting it to a list of floats if necessary.
• __call__(): This is the main entry point for the FeatureExtractionPipeline, which calls the other methods to perform the end-to-end feature extraction.

The FeatureExtractionPipeline is designed to provide a concise and efficient way to extract features from transformer models, without requiring users to handle the low-level details of the feature extraction process.

Video Pipelines

References: src/transformers/pipelines/video_classification.py

The VideoClassificationPipeline class in the file …/video_classification.py is responsible for performing video classification tasks using pre-trained models from the Transformers library.

The key functionality of the VideoClassificationPipeline class includes:

• Checking if the required av backend is available and ensuring the input model is of the correct type (MODEL_FOR_VIDEO_CLASSIFICATION_MAPPING_NAMES).
• Handling the preprocessing and postprocessing parameters, such as top_k, num_frames, and frame_sampling_rate.
• Preprocessing the input video(s) by downloading the video if the input is a URL, opening the video container using av.open(), selecting the desired number of frames based on the num_frames and frame_sampling_rate parameters, and converting the video frames into a format suitable for the model using the image_processor.
• Passing the preprocessed input to the model and obtaining the model's output.
• Processing the model's output and converting it into a human-readable format, extracting the top top_k scores and their corresponding labels.
• The read_video_pyav() function is a helper function used by the preprocess() method to read the video frames from the av container.

Utilities

References: src/transformers/utils

The Transformers library provides a range of utility functions and classes that are used throughout the codebase. These utilities cover functionality including:

• Tensor and Array Manipulation: The …/generic.py file contains utility functions for working with tensors and NumPy arrays. It includes functions such as is_tensor, to_numpy, transpose, torch_int, and torch_float. The torch_int function casts the input to a PyTorch int64 tensor if the input is being traced, otherwise it casts it to a Python int. The torch_float function casts the input to a PyTorch float32 tensor if the input is being traced, otherwise it casts it to a Python float.

• Enumerations and Custom Classes: The …/generic.py file also defines several enumerations and custom classes, such as ExplicitEnum, PaddingStrategy, TensorType, and ModelOutput, which are used throughout the library.

• Miscellaneous Utilities: The …/generic.py file provides helper functions for tasks like working with directories, parsing environment variables, and handling enumerations, such as working_or_temp_dir and strtobool. A decorator function filter_out_non_signature_kwargs exists to filter out named arguments that are not in the function's signature, issuing a warning for any invalid keyword arguments.

• Backbone Utilities: The …/backbone_utils.py file contains functionality for handling backbones in Transformers models. It includes methods to initialize the backbone from either the TIMM library or the Transformers library, and properties to access and set the out_features and out_indices configurations. The load_backbone function raises a ValueError if both backbone_config and backbone are specified, and a function verify_backbone_config_arguments verifies the validity of the config arguments passed to load_backbone.

• 8-bit Quantization: The …/bitsandbytes.py file provides functions for enabling 8-bit quantization of transformer models, which can reduce the memory footprint and inference time of the models.

• Chat Template Generation: The …/chat_template_utils.py file provides functionality for generating JSON schemas from Python functions, which can be used in chat templates that support tool integration. It includes functions for rendering chat templates with assistant indices, compiling Jinja templates, and tracking assistant-generated tokens in the rendered chat.

• Logging and Notebook Utilities: The …/logging.py and …/notebook.py files contain utility functions and classes for configuring the library's root logger and displaying progress bars and metrics in a Jupyter Notebook or Google Colab environment.

• Model Parallel Utilities: The …/model_parallel_utils.py file contains functions for managing device mappings and ensuring the correct configuration of attention blocks in a model-parallel setup.

• Quantization Configurations: The …/quantization_config.py file defines various quantization configuration classes, which provide an interface for configuring different quantization methods.

These utility functions and classes provide a range of functionality that is used throughout the codebase. For more information on specific aspects of the utilities, please refer to the corresponding sections in the wiki:

Compatibility and Dependency Management File and Cache Management Docstring and Type Hint Handling Tensor and Array Manipulation Backbone Utilities 8-bit Quantization Chat Template Generation Logging and Notebook Utilities Model Parallel Utilities Quantization Configurations

Compatibility and Dependency Management

References: src/transformers/utils/import_utils.py

The …/import_utils.py file contains utility functions and variables related to importing various libraries and packages required by the Transformers library.

The file defines a set of functions to check the availability of various packages, such as is_torch_available(), is_tf_available(), and is_flax_available(). These functions use the _is_package_available() function to check if a package is installed and its version.

The file also reads environment variables like USE_TF, USE_TORCH, and USE_FLAX to determine the preferred deep learning framework to use. It defines constants for the minimum required versions of various packages, such as ACCELERATE_MIN_VERSION and FSDP_MIN_VERSION.

The file provides several helper functions, such as is_torch_deterministic(), is_torch_sdpa_available(), and is_torch_cuda_available(), to abstract away the complexity of checking the availability and capabilities of the PyTorch and TensorFlow backends.

The file also defines a set of error messages that are displayed when a required package is not found, and the requires_backends() function is used to raise these error messages. The DummyObject metaclass is used to create classes that raise the appropriate error message when accessed.

The _LazyModule class is used to lazily load modules and objects, and the lru_cache() decorator is used to cache the results of certain functions to improve performance.

Docstring and Type Hint Handling

References: src/transformers/utils/doc.py

The main functionality in the file …/doc.py includes:

• Docstring Manipulation Functions:

  • add_start_docstrings(): Adds start docstrings to a function.
  • add_start_docstrings_to_model_forward(): Adds start docstrings to the forward() method of a model.
  • add_end_docstrings(): Adds end docstrings to a function.
  • replace_return_docstrings(): Replaces the "Returns:" section of a function's docstring with a prepared output docstring.

• Output Docstring Preparation:

  • _prepare_output_docstrings(): Prepares the "Returns:" section of a function's docstring based on the output type and configuration class.
  • _convert_output_args_doc(): Converts the output arguments documentation to display properly.

• Sample Code Snippets:

  • The file contains various sample code snippets for different model tasks and architectures, such as PT_SEQUENCE_CLASSIFICATION_SAMPLE, TF_SPEECH_CTC_SAMPLE, etc. These snippets are used to add example usage to the model docstrings.

• Utility Functions:

  • _get_indent(): Returns the indentation in the first line of a given text.
  • filter_outputs_from_example(): Removes the lines testing an output with the doctest syntax in a code sample when it's set to None.
  • copy_func(): Returns a copy of a function.

Tensor and Array Manipulation

References: src/transformers/utils/generic.py

The …/generic.py module in the Transformers library provides a set of utility functions for working with tensors and NumPy arrays. These functions simplify common operations and provide a consistent interface across different deep learning frameworks.

The key functionality includes:

• Tensor Utilities:

  • is_tensor(): Checks if an object is a tensor.
  • to_numpy(): Converts a tensor to a NumPy array.
  • to_py_obj(): Converts a tensor or NumPy array to a Python object.
  • infer_framework_from_repr(): Infers the deep learning framework from the representation of a tensor.
  • _get_frameworks_and_test_func(): Retrieves the available deep learning frameworks and a function to test if an object is a tensor.
  • torch_int(): Casts the input to a PyTorch int64 tensor during tracing, otherwise to a Python int.
  • torch_float(): Casts the input to a PyTorch float32 tensor during tracing, otherwise to a Python float.

• ModelOutput Class:

  • ModelOutput: A base class for model outputs that provides a consistent interface for accessing model predictions.

• Enumerations:

  • ExplicitEnum: A custom enumeration class that provides more explicit string representations.
  • PaddingStrategy: An enumeration for different padding strategies.
  • TensorType: An enumeration for different tensor types.

• Utility Functions:

  • can_return_loss(): Checks if a model output can return a loss value.
  • find_labels(): Finds the labels in a model output.
  • flatten_dict(): Flattens a nested dictionary.
  • working_or_temp_dir(): Returns a working directory or a temporary directory.
  • transpose(), reshape(), squeeze(), expand_dims(): Tensor manipulation functions.
  • tensor_size(): Returns the size of a tensor.
  • add_model_info_to_auto_map(), add_model_info_to_custom_pipelines(): Utility functions for managing model information.
  • infer_framework(): Infers the deep learning framework from an object.
  • filter_out_non_signature_kwargs(): Filters out named arguments not in the function's signature, and issues a warning for any invalid keyword arguments.

These utility functions provide a consistent and convenient way to work with tensors and arrays across different deep learning frameworks, simplifying the development of Transformer-based models and applications.

Logging and Notebook Utilities

References: src/transformers/utils/logging.py, src/transformers/utils/notebook.py

The …/logging.py file contains utility functions and classes for configuring the library's root logger and controlling the display of progress bars and metrics in a Jupyter Notebook or Google Colab environment.

The main functionality includes:

• _configure_library_root_logger(): Sets up the root logger for the Transformers library, including the default log level and formatting.
• get_logger(): Retrieves a logger instance for a given module or name.
• set_verbosity(), disable_default_handler(), enable_default_handler(), add_handler(), remove_handler(), disable_propagation(), enable_propagation(), enable_explicit_format(), reset_format(): Functions for managing the behavior of the root logger.
• warning_advice(), warning_once(): Utility functions for issuing warnings.
• EmptyTqdm: A custom progress bar class that does not display any output, used when progress bars are disabled.
• is_progress_bar_enabled(), enable_progress_bar(), disable_progress_bar(): Functions for controlling the display of progress bars.

The …/notebook.py file provides additional utilities for displaying progress bars and metrics in a Jupyter Notebook or Google Colab environment. The key components are:

• format_time(), html_progress_bar(), text_to_html_table(): Utility functions for formatting time, generating HTML progress bars, and converting text to HTML tables.
• NotebookProgressBar: A class that displays a progress bar in a notebook, with customizable behavior.
• NotebookTrainingTracker: A class that extends NotebookProgressBar to display a table of metrics alongside the progress bar.
• NotebookProgressCallback: A TrainerCallback that integrates the NotebookTrainingTracker and NotebookProgressBar classes to provide a comprehensive training progress display in a notebook.

File and Cache Management

References: src/transformers/utils/hub.py

The …/hub.py file provides a set of utilities for handling file caching and management, crucial for the efficient operation of the Transformers library. It includes functions for downloading and caching model files, which are pivotal for users who need to work with pre-trained models offline or want to avoid re-downloading models multiple times.

• The cached_file() function serves as the primary method for retrieving model files, either by downloading them from the Hugging Face Hub or by fetching them from a local cache if they have already been downloaded. This function is designed to streamline the process of working with model files, ensuring that users have quick and easy access to the models they need.

• The http_user_agent() function generates a user-agent string that includes details about the Transformers library version, the Python version, and the versions of underlying machine learning frameworks like PyTorch or TensorFlow. This information is typically used in HTTP requests when interacting with the Hugging Face Hub.

For users looking to contribute to the Hugging Face Hub, the PushToHubMixin class provides methods to facilitate this process. It includes the following key functionalities:

• The _create_repo() method, which allows for the creation of a new repository on the Hub, enabling users to share their models with the community.

• The _get_files_timestamps() method, which retrieves the last modification timestamps of files in the working directory, a feature that helps in determining which files have been updated and need to be pushed to the Hub.

• The _upload_modified_files() method, which handles the uploading of modified files to a specified repository on the Hub. This method ensures that only the necessary files are uploaded, optimizing the push process.

• The push_to_hub() method is the central function for pushing models, tokenizers, and other objects to the Hugging Face Hub. It abstracts away the complexity of the upload process, providing a user-friendly interface for sharing work with the community.

Additionally, the file includes functions for telemetry and integration with cloud services:

• The define_sagemaker_information() function collects information about the Amazon SageMaker environment, which can be useful for users running the Transformers library in the cloud.

• The send_example_telemetry() function is responsible for sending telemetry data related to the usage of Transformers examples. This data helps the library maintainers understand how the examples are being used and can guide future improvements.

Lastly, the move_cache() function addresses the need for cache migration by moving cached files to a new directory structure. This utility is essential when changes to the caching mechanism are made, ensuring that users' cached files remain accessible and organized.

For more information on how these utilities are used within the Transformers library, refer to the Testing section, which discusses the testing framework that ensures the reliability of these file and cache management utilities.

Dummy Objects for Backend Compatibility

References: src/transformers/utils/dummy_pt_objects.py, src/transformers/utils/dummy_vision_objects.py

In the Transformers library, backend compatibility is maintained through the use of dummy classes, which act as stand-ins for required functionalities when certain dependencies are not installed. These dummy classes are found in …/dummy_pt_objects.py and …/dummy_vision_objects.py, and they prevent import errors that would otherwise occur due to missing backends.

• Classes like DummyObject are defined to represent components from external libraries such as PyTorch. When an instance of a dummy class is created, the requires_backends() function is invoked to check for the availability of the necessary backend. If the backend is not present, an error is raised, alerting the user to the missing dependency.

• Vision-related dummy objects are provided to offer a consistent API for image processing tasks. Classes such as ImageProcessingMixin, BaseImageProcessor, and ImageFeatureExtractionMixin require a "vision" backend to function. These classes are designed to interact seamlessly with the rest of the Transformers library, ensuring that the absence of vision-related libraries does not disrupt the overall workflow.

• Specific dummy classes for various vision models, like CLIPFeatureExtractor and BeitImageProcessor, mimic the behavior of their respective feature extractors and image processors. These classes inherit from DummyObject and are essential for maintaining the library's functionality across different environments, regardless of whether the required vision backends are installed.

By implementing these dummy classes, the Transformers library provides a robust solution for handling backend dependencies, allowing users to work with the library in diverse setups without encountering compatibility issues.

Configuration and Processor Mapping Utilities

References: src/transformers/models/auto/configuration_auto.py, src/transformers/models/auto/processing_auto.py, src/transformers/models/auto/image_processing_auto.py, src/transformers/models/auto/modeling_auto.py, utils/check_config_attributes.py

The …/configuration_auto.py file contains the AutoConfig class, which serves as a gateway for instantiating configuration objects for various pre-trained models. It leverages a mapping system that associates model identifiers with their respective configuration classes. When a user requests a configuration for a specific model, AutoConfig dynamically loads the appropriate class based on this mapping. This mechanism supports the addition of new model types by updating the mapping dictionary, allowing for easy expansion of the library's capabilities.

In …/processing_auto.py, the AutoProcessor class operates similarly to AutoConfig, but for processors. It uses the PROCESSOR_MAPPING_NAMES and PROCESSOR_MAPPING to associate model types with their corresponding processor classes. The processor_class_from_name() function retrieves the correct processor class given a class name. The AutoProcessor.register() method allows for the registration of new processors, facilitating the integration of custom or newly developed processors into the Transformers ecosystem.

The …/image_processing_auto.py file extends these capabilities to image processing. It introduces the AutoImageProcessor class, which uses the IMAGE_PROCESSOR_MAPPING_NAMES and IMAGE_PROCESSOR_MAPPING to link configuration classes with their corresponding image processors. The image_processor_class_from_name() function and AutoImageProcessor.register() method enable dynamic retrieval and registration of image processors, respectively.

For model instantiation, …/modeling_auto.py provides a suite of AutoModel classes, such as AutoModelForPreTraining and AutoModelForCausalLM, which use mappings like MODEL_MAPPING to automatically select the correct model class. These classes streamline the process of working with different model architectures by abstracting away the need to specify the exact model class.

Lastly, …/check_config_attributes.py includes functions like check_attribute_being_used() and check_config_attributes_being_used() to verify that all attributes defined in a configuration class's __init__ method are actually utilized in the corresponding modeling files. This utility ensures that the configuration classes remain aligned with the model implementations, preventing discrepancies that could lead to errors or inefficiencies.

These utilities collectively support the Transformers library's extensibility and maintainability, allowing developers to manage configurations and processors for a growing number of models without modifying the core logic of the library.

Repository Checking and Validation

References: utils/check_repo.py

Maintaining the integrity and consistency of the Transformers repository is crucial for developers and users alike. A set of utilities within …/check_repo.py facilitates this by performing a series of checks across the repository:

• The check_model_list() function compares the subfolders under …/models with the models registered in the transformers.models submodule to ensure that the list of models is accurate and up to date.

• With check_models_are_in_init(), the script verifies that all models defined in the library are included in the main __init__.py, which is essential for correct imports and usage within the library.

• The check_all_models_are_tested() function, alongside get_model_test_files() and check_models_are_tested(), ensures that each model has corresponding test cases, which is vital for validating model functionality and reliability.

• Documentation is verified through check_all_objects_are_documented() by matching documented objects found by find_all_documented_objects() against those in the main __init__.py, promoting comprehensive documentation practices.

• Auto configuration mappings are validated using check_all_models_are_auto_configured() and get_all_auto_configured_models(), confirming that models are correctly set up for dynamic loading through the library's auto classes.

• The functions check_all_auto_object_names_being_defined() and check_all_auto_mapping_names_in_config_mapping_names() ensure that all names in auto mappings are defined and match those in CONFIG_MAPPING_NAMES, which is crucial for the auto configuration system's integrity.

• Importability of auto mappings is checked by check_all_auto_mappings_importable(), which prevents issues with missing or incorrect imports that could disrupt the user experience.

• Cross-framework consistency is maintained with check_objects_being_equally_in_main_init(), which checks for the presence of objects like TensorFlow or Flax versions of a model in the main __init__.py.

• The check_deprecated_constant_is_up_to_date() function ensures that the DEPRECATED_MODELS constant reflects the current state of model deprecations, aiding in proper deprecation management.

The check_repo_quality() function serves as the entry point to execute all these checks, providing a systematic approach to repository validation.

Quantization Configuration Handling

References: src/transformers/utils/quantization_config.py

The Transformers library facilitates the application of various quantization methods to models through a suite of configuration classes located in …/quantization_config.py. These classes enable users to tailor the quantization process to their specific needs by setting parameters unique to each quantization technique. The library supports a range of methods, including Bits and Bytes, GPTQ, AWQ, AQLM, Quanto, EETQ, HQQ, Compressed Tensors, FBGEMM FP8, and TorchAO.

• QuantizationConfigMixin serves as the foundational class, providing serialization and deserialization capabilities, which are essential for saving and loading quantization configurations. It also includes an update() method, allowing for dynamic adjustments to the configuration during runtime.

• Each quantization method has a dedicated configuration class, such as HqqConfig for HQQ, which encapsulates parameters like bit precision and group size. These classes include post_init() methods that perform validation checks to ensure the provided settings are within acceptable ranges and compatible with the quantization library in use.

• The BitsAndBytesConfig class supports advanced quantization techniques like LLM.int8() and FP4/NF4, with properties ensuring that 4-bit and 8-bit quantization options are not simultaneously active.

• GPTQConfig is tailored for large language models, integrating parameters for tokenizer, dataset, and quantization specifics. It validates compatibility with the optimum library, which is crucial for applying GPTQ to models.

• AwqConfig and AqlmConfig handle Adaptive Weight Quantization and Additive Quantization for Linear Mappings, respectively. They include options for fine-tuning the quantization granularity and excluding specific modules from the quantization process.

• QuantoConfig and EetqConfig provide configurations for Quanto and Efficient Embedded Tensor Quantization methods, focusing on data types for weights and activations and allowing exclusions for certain modules.

• CompressedTensorsConfig deals with the storage of quantized model checkpoints, offering detailed settings for compression ratios, sparsity, and cache schemes. It features custom methods for handling nested configurations.

• FbgemmFp8Config and TorchAoConfig wrap around FBGEMM FP8 and TorchAO's quantization and sparsity techniques, providing options for activation scales and quantization types.

Deprecated parameters are managed within these configuration classes, ensuring backward compatibility and smooth transitions to updated quantization features. The design of these classes allows for the easy addition of new quantization methods and features, maintaining the library's adaptability to evolving quantization techniques.

For more information on the utility functions and classes used throughout the Transformers library, refer to the Utilities section.

Examples

References: examples

The examples directory contains a comprehensive set of examples and utilities for fine-tuning and evaluating various Transformer-based models on a wide range of machine learning tasks, including natural language processing, computer vision, and speech recognition.

The examples are organized into several subdirectories, each focusing on a specific task or use case:

• The …/pytorch directory provides a comprehensive set of examples and utilities for fine-tuning and evaluating Transformer-based models using PyTorch. This includes scripts for tasks such as text classification, question answering, language modeling, and speech recognition.
• The …/tensorflow directory contains scripts and utilities for fine-tuning Transformer-based models using the TensorFlow library. This includes examples for tasks like image classification, question answering, and text summarization.
• The …/research_projects directory contains code and scripts related to the Retrieval-Augmented Generation (RAG) models, which combine a question encoder and a document retriever.